feat: add tests for guide code snippets by m4tx · Pull Request #552 · cot-rs/cot

m4tx · 2026-04-12T21:45:22Z

This adds a test framework to test the code snippets in the guide (the docs/ directory). Currently, the framework defines four different test configurations:

toml, tested by parsing as a Cot config file
html.j2, tested by trying to build an Askama template
rust, tested by building the snippet as an extension of a default project template (as generated by cot-cli)
rust,has_main, a version of the above that also defines its own main function

It should be fairly easy to extend this by defining more configurations, if needed.

The framework defines a number of utilities to make the testing more pleasant:

Many symbols are imported by default
Similarly to rustdoc, lines can be prepended with # to indicate lines that are not displayed in the guide, but required for the test to build
For the Askama tests, there are some dummy files and structures created

This is part of the major effort to make the guide always up-to-date. As a result of this, a number of code snippets were modified as they did not build before.

This concludes the work on #471.

Type of change

Copilot

Pull request overview

Adds an integration-test framework to compile/check code blocks in the docs/ guide, and updates multiple guide snippets so they build under the new framework.

Changes:

Introduces cot-test crate + libtest-mimic runner to validate fenced code blocks in docs/*.md (Rust/TOML/Askama templates).
Updates many guide snippets (hidden # ... lines, async fixes, etc.) to compile under the new checking approach.
Adds supporting glue: just test-docs, nextest concurrency override, and small framework/API touch-ups (e.g., TestRequestBuilder::cache, ForeignKey::new).

Reviewed changes

Copilot reviewed 25 out of 27 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
justfile	Adds `test-docs` task/alias to run the new doc-snippet tests.
docs/upgrade-guide.md	Adjusts a Rust code fence and list indentation.
docs/testing.md	Updates examples; adds async fixes; minor formatting changes.
docs/templates.md	Updates Askama examples and adds extra supporting code for compilation.
docs/static-files.md	Adds hidden lines so static-file snippets compile.
docs/site/Cargo.lock	Bumps `cot-site` git rev in the docs site lockfile.
docs/README.md	Reworks headings and adds placeholder section for doc-test instructions.
docs/openapi.md	Updates OpenAPI examples (return types, derives, route registration) to compile.
docs/introduction.md	Adds hidden scaffolding + a `rust,has_main` snippet for the `main` example.
docs/forms.md	Updates form examples to use newer APIs (`into_response()?`) and adds hidden scaffolding.
docs/db-models.md	Makes DB examples compile (adds hidden model scaffolding; comments out sample URLs).
docs/caching.md	Adds hidden scaffolding so cache examples compile.
docs/admin-panel.md	Updates admin-panel snippets for newer APIs and adds missing scaffolding.
cot/src/test.rs	Adds `TestRequestBuilder::cache()` behind `cache` feature.
cot/src/project.rs	Updates a panic message to reference `cot::Project`.
cot/src/db/relations.rs	Adds `ForeignKey::new(...)` convenience constructor.
cot-test/tests/doc_code_blocks.rs	New libtest-mimic runner that parses Markdown and runs per-block checks.
cot-test/templates/html_main.rs	Template-compilation harness for Askama snippets.
cot-test/templates/base.html	Base template used by Askama snippet compilation.
cot-test/src/lib.rs	New helper crate that generates a temp project and runs `cargo check` for snippets.
cot-test/Cargo.toml	Defines new internal crate and custom test harness entry.
cot-macros/tests/ui/func_query_invalid_field.stderr	Updates expected UI test stderr output wording.
cot-macros/tests/ui/attr_main_args.stderr	Updates expected stderr to reference `cot::Project`.
cot-macros/tests/ui/attr_main_args.rs	Updates return type to `cot::Project`.
Cargo.toml	Adds `cot-test` to workspace + new workspace deps (`comrak`, `libtest-mimic`, etc.).
Cargo.lock	Updates lockfile for new deps and `cot-test` crate.
.config/nextest.toml	Adds nextest grouping to serialize doc-code-block tests.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

github-actions · 2026-04-12T21:52:15Z

Bencher Report

Branch	docs-tests-2
Testbed	github-ubuntu-latest

Click to view all benchmark results

Benchmark	Latency	Benchmark Result microseconds (µs) (Result Δ%)	Upper Boundary microseconds (µs) (Limit %)
empty_router/empty_router	📈 view plot 🚷 view threshold	5,757.40 µs (-5.27%) Baseline: 6,077.48 µs	7,391.53 µs (77.89%)
json_api/json_api	📈 view plot 🚷 view threshold	1,027.30 µs (-3.19%) Baseline: 1,061.12 µs	1,264.88 µs (81.22%)
nested_routers/nested_routers	📈 view plot 🚷 view threshold	985.79 µs (+1.05%) Baseline: 975.52 µs	1,153.86 µs (85.43%)
single_root_route/single_root_route	📈 view plot 🚷 view threshold	961.72 µs (+2.51%) Baseline: 938.20 µs	1,118.80 µs (85.96%)
single_root_route_burst/single_root_route_burst	📈 view plot 🚷 view threshold	19,807.00 µs (+12.03%) Baseline: 17,679.47 µs	21,082.66 µs (93.95%)

🐰 View full continuous benchmarking report in Bencher

This adds a test framework to test the code snippets in the guide (the docs/ directory). Currently, the framework defines four different test configurations: * `toml`, tested by parsing as a Cot config file * `html.j2`, tested by trying to build an Askama template * `rust`, tested by building the snippet as an extension of a default project template (as generated by `cot-cli`) * `rust,has_main`, a version of the above that also defines its own `main` function It should be fairly easy to extend this by defining more configurations, if needed. The framework defines a number of utilities to make the testing more pleasant: * Many symbols are imported by default * Similarly to rustdoc, lines can be prepended with `# ` to indicate lines that are not displayed in the guide, but required for the test to build * For the Askama tests, there are some dummy files and structures created This is part of the major effort to make the guide always up-to-date. As a result of this, a number of code snippets were modified as they did not build before. This concludes the work on #471.

m4tx · 2026-04-13T18:26:12Z

Note: this PR still needs a corresponding one in cot-site for hiding rust lines starting with # , but this one is ready for review.

Copilot

Pull request overview

Copilot reviewed 23 out of 25 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

codecov · 2026-04-13T19:06:47Z

Codecov Report

❌ Patch coverage is 74.61538% with 33 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
cot-test/src/lib.rs	76.80%	7 Missing and 22 partials ⚠️
cot/src/test.rs	0.00%	4 Missing ⚠️

Flag	Coverage Δ
rust	`89.89% <74.61%> (-0.10%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
cot/src/project.rs	`88.76% <100.00%> (ø)`
cot/src/test.rs	`85.41% <0.00%> (-0.66%)`	⬇️
cot-test/src/lib.rs	`76.80% <76.80%> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ElijahAhianyo · 2026-04-16T00:12:32Z

+                // For specific symbols, only add if the symbol is NOT already imported
+                let mut found = false;
+                for line in code.lines() {
+                    if line.starts_with("use ") && line.contains(symbol_name) {


Will this handle cases where symbol names are aliased and not necessarily what we're looking for?

Eg, use cot::common_types::Email and use cot::email::Email as EmailService. If we have common_types::Email in COMMON_IMPORTS, then use cot::email::Email as EmailService could still be a match. No?

ElijahAhianyo · 2026-04-16T00:15:23Z

+        }
+
+        let final_code = match test_type {
+            RUST_HAS_MAIN_TEST_TYPE => code.to_string(),


Shouldn't we use an enum for this in case we add more variants in the future?

ElijahAhianyo · 2026-04-16T00:46:50Z

        self
    }

+    /// Add a cache to the request builder.


wondering if we'll also need to add one for email

seqre · 2026-05-02T15:12:09Z

+    test_runners.insert(("rust", DEFAULT_TEST_NAME), test_rust);
+    test_runners.insert(("rust", RUST_HAS_MAIN_TEST_TYPE), test_rust);
+    test_runners.insert(("toml", DEFAULT_TEST_NAME), test_toml);
+    test_runners.insert(("html.j2", DEFAULT_TEST_NAME), test_html);


I think I'd like an enum for language options too

m4tx requested review from ElijahAhianyo, Copilot and seqre April 12, 2026 21:45

github-actions Bot added A-docs Area: Documentation A-deps Area: Dependencies C-lib Crate: cot (main library crate) C-macros Crate: cot-macros labels Apr 12, 2026

Copilot started reviewing on behalf of m4tx April 12, 2026 21:45 View session

m4tx force-pushed the docs-tests-2 branch 2 times, most recently from 804f216 to 352ef6f Compare April 12, 2026 21:48

m4tx removed request for ElijahAhianyo and seqre April 12, 2026 21:48

m4tx marked this pull request as draft April 12, 2026 21:48

Copilot AI reviewed Apr 12, 2026

View reviewed changes

Comment thread cot-test/src/lib.rs

Comment thread docs/introduction.md

Comment thread docs/README.md Outdated

Comment thread Cargo.toml Outdated

Comment thread docs/testing.md Outdated

Comment thread docs/templates.md

Comment thread docs/templates.md

m4tx changed the title ~~feat: add tests for guide code snippet~~ feat: add tests for guide code snippets Apr 13, 2026

m4tx force-pushed the docs-tests-2 branch 4 times, most recently from 75dc48c to 5b957f4 Compare April 13, 2026 18:19

m4tx marked this pull request as ready for review April 13, 2026 18:20

m4tx force-pushed the docs-tests-2 branch from 5b957f4 to 192175f Compare April 13, 2026 18:23

m4tx requested review from ElijahAhianyo, Copilot and seqre April 13, 2026 18:26

Copilot started reviewing on behalf of m4tx April 13, 2026 18:27 View session

Copilot AI reviewed Apr 13, 2026

View reviewed changes

Comment thread cot-test/src/lib.rs Outdated

Comment thread cot-test/src/lib.rs Outdated

ElijahAhianyo reviewed Apr 16, 2026

View reviewed changes

m4tx mentioned this pull request Apr 27, 2026

feat: hide code block lines prepended with # cot-rs/cot-site#87

Merged

seqre requested changes May 2, 2026

View reviewed changes

address review comments

93c2ff9

Uh oh!

Conversation

m4tx commented Apr 12, 2026

Type of change

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bencher Report

Uh oh!

m4tx commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ElijahAhianyo Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

ElijahAhianyo Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ElijahAhianyo Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

seqre May 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions Bot commented Apr 12, 2026 •

edited

Loading

m4tx commented Apr 13, 2026 •

edited

Loading

codecov Bot commented Apr 13, 2026 •

edited

Loading